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ABSTRACT 

The handbook 'ils .intended to assist evaluators working 
with preschool model demonstration projects funded by the Handicapped 
Children's Early Education Program (HCEEP) . Background on HCEEP is 
provided, and the- function of evaluation within the network is 
emphasized. A section on planning evaluations deals with* such aspects 
as focusing, planning specific evaluation strategies, considering 
evaluation methods,, selecting instrumentation, and planning for 
reporting and implementing the evaluation. The roles of the Joint 
Dissemination Review Panel, designed to assess the effectiveness of 
federally funded programs, are- considered with special emphasis on 
the relationship of HCEEP and the review panel. A final section 
presents suggestions ©a procedural and personal matters for ' 
evaluation consultantsV (CL) 
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' INTRODUCTION 

4 

This Handbook provides descriptive information, suggestions, and 
reference materials for internal evaluators, evaluation consultants, and 
external' ^valuators who )W ork with preschool model demonstration projects 
^ funded by the Handicapped Children's Early Education Program (HCEEP) . 
The purpose's of the Handbook are: ' 

, 1.) to provide a description of HCEEP demonstration projects and 
some basic information concerning the concept of and activities for. 
the model development for which they are responsible; 

2.) to describe evaluation needs and. planning considerations for 
■ the projects; ' • 

t * 

• vmp^ t0 P<r0Vide information on the Joint -Dissemination Review Panel ' ' 
XJDRP) as it relates to HCEEP: and 

. 4.) to describe 'consultation strategies and provide suggestions and 
hints for evaluation consultants when they are working with HCEEP 
- projects. 

It is important to specify W wha t the Handbook is not: it is not 
a textbook on program evaluation, nor a manual for' total program evaluation 
design. It does not .contain a complete guide to instrumentation for, 
documenting child or.parent change, hor does it 'provide data analysis , 

^guidelines. Each of those topics has been covered previously in many- ' 
excellent manuals and books, and a list of selected references- in-each. 
area is provided in the Appendix f6,r interested • readers . 

The Handbook, instead, is intended to be used td help evaluation 
consultants understand the context of evaluation in HCEEP , -to plan thorough, 
yet realistic^evaluations, to understand the -purpose and function of JD£P 
as -it relates to HCEEP projects, and to utilize their' evaluation skills 

.productively in their rel^io^lp*wJ«^|)rojects. 
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t EVALUATION IN HCEEP — TJJEjCONTEXT 

f ' ' ' 

Introduction 1 i 

The Handicapped Children's Early Education Program (HCEEP) began as' 
a small^rogram of 24 grants, for demonstration projects in 1969. The . 
program was designed to: a.) 'promote the development of locally-designed ' - 
ways to serve handicapped young children and their families; b.) gather 
•information on effective programs and techniques; and,c.) distribute 
visible, replicable models for services to this population throughout the 
country. ( 

Over the years the program .has expanded to meet the emerging needs of 
this fc field. HCEEP is nag, comprised of approximately 200 projects in -five 
component areas (Figure 1). As described in the figure, the technical 
assistance centers provide assistance to demonstration projects and state 
implementation grantees. Technical^ assistance in evaluation, the topic 
of this Handbook, is provided primarily to demonstration projects. 

The Demonstration Projects 1 
During '1979-80, the then»Bureau of Education for the Handicapped (now 
. the Office of Special Education [OSE] in the new U.S. Department , of Education*,' 
funded 127 three year demonstration proj'ects across the country. Of these 
35 were in their first year, 39 were in their second and 53 were in their 
third year of operation. While varied in many ways, these and other 
demonstration projects that have been funded through HCEEP focus on 



1 * • ■ r 

Portions of this introduction were tak eT r-from The 1979-80 Overview and 
girectorg jgr the Handicapped Children's Early EdujTjj^Prn^. Seattle, 
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development and services in four major areas: 1.) services for children; » 
2.) services for. parents; 3.) 'Staff development; and 4.) demonstration/ 

dissemination. ' 

• - , r , + 

Services for Children 

Approximately 3,600 children were served by the projects during 1979-^0, 
As in prevkou* years services were developed^ and provided" to .children with 
a variety of handicapping Conditions. Over 25% of 'the children served 

w^re multihandicapped. Host of the children served were between the ages 
• • * * 

0^,0-5. Recent initiatives have expanded the number of programs serving 
the youngest (ages 0-2 years) and more Severely handicapped children.' 
Depending on the type of program they have, proposed projects may serve 
single age groups, "e.g. infants, or multia^fd groups. . They may also serve ' 
children with the same handicapping conditio* or a variety of«*iandicapping 

conditions. ' • 

>. » 

The services provided to children also vary. The ^tirnary philosophical 

base for the services spans a continuum from child-centered* {experiential)* 

to teaser-directed (behavioral) . learning. Approximately half of the 

projects describe their approach as "diagnostic-prescriptive. 11 The focus 

y 

of services is ^>n child development in the area of language-communication, 
f sensorimotor, social-emotional, cognitiVe-academic and self-help development. 

In keeping with Public Law 94-142, educational services for the children 

\ ■* « 

■are individualized, and described in each child's Individualized Education 
Plan (IEP). Services are provided, again depending on the program, in the 
Home, at centers, in public schools and hospital and health care centers. 
Services for Parents - 

Each HCEEP demonstration project must provide services for the parents 
of the children that are served. The services may be. designed to encourage 
' V r - * 
O . f i 
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parents to become Involved with the project 'in 'a number of ways, .such as 
serving as volunteers, informing the public or other agencies of the needs 
of handicapped children, or attending parent group, meetings. In other 
projects, the parents may be the target for direct- project services, such 
as learning to be their child's teacher, or learning successful parenting 
techniques ; Parents also are involved with the development of .the IEP 
for their child, and frequently se.rve on project advisory councils. 
Staff Development 

. A third program component which is included in most project plans is 
the professional development of 'the project staff. Recognizing that the 
varied demands of model program development may require many new skills, 
projects are encouraged to include professional development funds in their 
budgets, and to develop a plan for the professional growth of each staff 



member. t * ^ 



Demonstration and Disse mifmHnn 

During t*e three years of federal, funding, HCEEP projects are expected 
to develop a program model and' to document its use. After the model is 
developed, projects demonstrate ' it to other interested professionals. 
Demonstration activities are frequently conducted at the project si'te, and 
involve showing and/or teaching others how to implement project activities. 

In addition, projects are involved in disseminating information ' 
concerning the project. ^Dissemination activities may include the development 
of brochures and slide-tap^s to be used for widespread dissemination of 
project information; personal, f^ace to face communication^ newspaper articles; 
• speeches -and presentations before prof e.ssion^ and lay groupsyand other ' 
similar activities. No matter what method is used, the purpose of 



dissemination is Co increase- awareness of the project services- and model', 
and to encourage others to consider replicating or adapting The model in 
order to increase services to young handicapped children. 

o ■ ( 

. The Concept of Model Development ' 
The previous description of the origin and purposes of HCEEP and of 
typical project program components serves as background information for 
understanding the notion o'f model development— the basic purpose and goal 
. of each demonstration proj-ect. . \ 1 

> 

■ While three years may sfeem like a long time when the grant ayard 

notiiication is «first received by the project, tA f act is that there is, 

much to be accomplished during this time. Model development is a complex- - 

and" challenging task. It is* important for evaluatf^^nsultants to be- , 

aware of the many and varied responsibilities and activities of the project 

staff in relation to model development and to assist them in designing an 

evaluation which will demonstrate the effectiveness of the specific S"*' 

. « /■ ' ' 

-important characteristics of their particular model. ' ( * 

^In order to assist project personnel and others to understand the 

mcdel "develdprnent process, TADS, 2 WESTAR, 3 arfd t)SE 4 have collaborated to 

• * * 

produce the Model Development Guid-e (Figure 2). The guide provides a 
task-by-tirae planning framework, outlining^Sour stages of program development 
(Planning, Implementation , Evaluation/Modificatio n and Maintenance). In 
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addition eight management tasks which are thought to be essential to the 
model development process are described. 

Use of the Guide assists projects in 1.) identifying and defining the 
basic elements of their model; 2.) documenting and evaluating the effects 
• of their efforts; and 3.) planning and initiating demonstration 'and 

replication activities. Evaluators will find the Model Development Guide 

'useful as a resource when assisting projects to develop useful and complete 
..program evaluation plans. 

In summary, the HCEEP demonstration projects are funded to develop 
^ and demonstrate innovative and effective models of service for young 

handicapped children and their families. In order to do so,* project staff 
-must be able to describe t£eir program and demonstrate i^s^alue. 'For 
these » reasons*, evaluation has been and remains a very important aspect of 
program development. The remainder of this Handbook addresses topics^ 
specifically related to the evaluation concerns of the projects and t^e 
ways in which evaluation - consultants can best be of assistance to project 
staffs. as they, plan and conduct evaluations of their programs. 
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PLANNING EVALUATIONS IN HCEEP 
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Introduction? 

Perhaps th^most conmum task which is asked of an evaluation consultant 
to HCEEP project is that of "assisting project staff members in the - 
development of an evaluation plan. Evaluations for HCEEP projects are ' 
conducted most often by.tfie project staff or by the project staff with 
• part time assistance of an evaluation consultant. The evaluations are, 
therefore, usually internal rather than extirnal in nature. 

In order for the project staff to conduct or supervise an evaluation, 
it is our opinion 'that 'they must know: a.) what they intend the evaluation 
to accomplish^ b.) what- specific evaluation strategies they, will employ; ' 
c.) how they will report and use evaluation results; and d.) how they will 
implement and manage the evaluation. 

To assist projects in planning in these areas, A Planning Guide for' 
the Evaluation of Educational Programs for Young Children and Their Famil ies 
(Suarez, 1980b) has been developed (Figure 4, located at the end of this . 
section of the Handbook .) The Planning Guide^Pflses a series of questions 
Co be answered in the four previously mentioned areas. They are labeled • 
in the Gu^de as: a.) fdcusing the evaluation; b.) determining the specific 
evaluation design; C .) communicating and using evaluation results; and d.) 
implementing the evaluation. (For additional information concerning specific 
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Portions of this Section were taken from Suarez, T.M. A Planning Guide for 
the Evaluation of "Education Programs for Young Children and Their FTrnlliisT 
Chapel Hill, N.C.: Technical Assistance Development System, 1980a (mimeo) 
and, Suarez, T.M. Evaluating Educational Programs^ for Preschool Handicapped 
Children. A paper presented to the annual AmericabTducational Research 
Association Meeting, Boston, April, 1980b. 



documentation guideline's, refer, to Suarez and Vandiviere, 1978.) 

We encourage project Staff members to be able, to answer the questions 
posed in the Guide. We do not, however, suggest that the-Guide is the format 
to be used for their plan. We, instead, encourage the use of formats that 
are clear and useful to the staff and consultant with whom they are working. 
m As can be seen in the instructions for using^ the Guide' , we hope that* 
_ it can be used for a variety of purposes. As the rbles and tasks of 
consultants are clarified; we hope that the Guide as a whole or in'p/t 
may also be useful in providing assistance to projects.' « • 

Commorients of Evaluation Planning 
• » ► 

The remainder of this chapter provides discussions of each of the 

major areas of the guide . They are intended to explain each question and 

provide consultants with information that can be used in assisting project 

staff members in planning their evaluations. \ 

Focusing ttfe Evaluation — ' ' . 

One of the more difficult tasks in planning an evaluation is to determine 
its focus. It demands the best diagnostic and process consultati^i skills 
of the evaluation consultant and a great deal of thinking, planning and' » 
decision making by the project staff. It req&res consideration of a 
variety of aspects of the program, e.g., its philosophy, and the skills 
and values of those most closely associated with it. . 

Evaluations have a focus when a.) the purposes for evaluation: b.) 
evaluation audiences and their needs; andc.) major aspects of the program 
to be evaluated are known. 

Project staff members know that one purpose of their evaluation is to 
report the effectiveness of their work to the Office of Special Education 
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(OSE). Beyond this purpose, intents for evaluations are not always specified 

Consultants can aid projects in better understanding thei/evaluations 
by assisting them in thinking about and specifying the purpose(s) for their 
evaluation. This clarification can aid a proj ect^T^cpanding their 
uses for evaluation, e.g. in conducting evaluations to improve their ' . *' 
program as well as meet federal accountability requirements, and to lay 
. the groundwork, for determining the contenflnd type of evaluation that is 
needed for their particular program. 

Consultants can be equally helpful to projects in helping the staffs 
expand their thinking about the audiences for their evaluation efforts. 
Common audiences for program evaluation for HCEEP projects are the funding 
agency (OSE)^ sources which might.be considered for continuation funding, 
project staff and parents. Helping the staff to identify specific audiences 
and their needs early in the planning process will assist in ensuring that' , 
they hav^fhe documentation and data that will be requested of them later. 
Perhaps the most dramatic illustration of this is the instance in which a 
principal audience for evaluation is tfie Joint Dissemination and Review 
Panel (JRDP). Information which is required for this particular audience 
.is specified in publications such as the JDRP Ideabook (Tallmadge, 1977). 
Failure- to consider this audience and its particular needs in advance may- 
lead to an evaluation which will not allow the project to be considered 
for JDRP review. 

A final consideration in focusing the evaluation is developing a clear 
picture of the project itself. This includes such activities as assisting 
the staff in identifying or clarifying the project's theoretical base, 
describing the key components or particular characteristics of the services ,'. 
to be provided, and delineating the project's intent or goals, i.e. 
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describing the project's model. Both the evaluators 1 and project personnel 

must have a clear understanding of what the program is and what it is 11 

trying to do before an appropriate evaluation can be designed. 

If a gocfd job is done in focusing the evaluation, the evaluator and 
> 

program personnel will have a clear idea of the direction in which the 

X 

design of the evaluation should go. Both the evaluator 1 " and the 'program 
staff will be able to state in concise terms the reasons for- conducting 
the evaluation, what it is they hope to get from the evaluation, and to 
whom they will provide the information. 
Planning Specific Evaluation Strategies 

Tnis is the-pa r t of the evaluation planning that is molt familiar to 
project itaff and evaluation consultants. It consists of determining the' 
evaluation questions, specifying the approach or design to be used in the 
evaluation, determining data collection and analysis prooedures and 
establishing criteria for judging the adequacy of the evaluation results. 
Discussions of these planning areas are presented here to point out some 
of the particular needs and concerns of HCEEP demonstration projects. 

In assisting staff members in this part of their evaluation plans, 
consultants should encourage the selection of procedures which are: 
a.) conceptually ^alid, i.e. they are appropriate for the content and 
procedures to which they are addressed; b.) logically "consistent , i.e. 
provide information relevant to evaluation intents and questions; c.) 
methodologically sound, i.e. use acceptable, established and/or logical 
procedures; and d.) feasible , i.e. do-able within the constraints of time, 
resources and expertise available for the task. 
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Evaluation Questions. Evaluation consultants, with their experience and 
••&*pertise in evaluation practice and methodology, cafi be especially 'helpful 
■to project staffs in the specif ication, of clear aa/ concise' questions to •* 
be addressed. They can help the staff relate the questions to be angered 
to the focus of the evaluation. They^can also help in the development^ 
questions which address the most important aspects of the Vrogram and 
prevent the development of too many or trivial questions. ^ 
Evaluation Methods, The most common evaluation-methods 'used by projects \ 
are the pre and post tests to determine change and the survey to describe \ 
knowledge, attitudes and opinions regarding a program. Evaluation 
consultants can be helpful to project staffs by 'considering and explaining 
other methods which may be equally or more appropriate. 

There are a variety of ways in which evaluation methods, designs or 
-models can be classified. As shown in Figure 3,'\pn« type of classification 
system of designs suitable for HCEEP projects is to divide them^nto those 
that are 1.) experimental, 2.) objective based, 3.) systems models, and 
4.) naturalistic. 

Experimental Designs. Among the more common evaluation designs which project 
staffs use are those which are experimental and quasi-experimental. 
Reference to the criteria of validity, logic, soundness of methodology and 
feasibility often reveals limitations in the use of these designs. For 
example, in order to maintain, the methodological soundness of designs one 
would not wish to use those which are considered to be uninterpretable 
(Cook and Campbell, 1979), i.e., the onej group post-t^st only design, the 
one group pre-test post-test design, and the post-test only design with 
non-equivalent groups. Other more appropriate designs are often not feasible 
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Figure 3 * 
Evaluation Designs for Programs for Preschool Handicapped Children 6 



Type of Design 



Experimental 



Objective Based 



Systems 



Purpose 



Examples 



To determine, the cause of observed outcomes 



Developmental designs (Porges, 1979) 
Experimental designs (Campbell & Stanley, 1963) 
Quasi-experimental designs (Cook & Campbell, 1979) 
Single-subject designs (Kratochvill, 1978) 



to determine if stated objective were achieved 
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(Popham, 1972) 4 

Program planning and evaluation model (Gallaeher 
Surles & Hayes, 1972) 

Goal-attainment scaling (Kiresuk & Lun*/ 1976; Carr, 1979) 



To^determine outcomes with the inclusion of » 
data regarding the relationship among ■ ^ M 
outcomes, inputs and processes 

processes Countenance Model (Stake, 1967) 



CIPP Evaluation Model (Stuff l«beam, et al.,19?l) 
Countenance Model (Stake, 1967) 
Discrepancy Model (Prpvus, 1971) 



Naturalistic 



To describe the program and its impacts 



Case study (Stake, 1978; Kennedy, 1978) 

Naturalistic inquiry (Guba, 1978; Wolf, 1979) 

Qualitative Evaluation (Willis, 1977) 
i 

Ethnography (Wilson, 1977) 



L'atarch^o^^ A pape'r presented to the African Educational 
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due to very small sample sizes, the inability to consider the cHildfen*in 
the program as a homogeneous group, the lack of adequate 'control groups 
and the often dramatic changes in treatment. 



In cases in which traditional experimental and quasi-experimental 
designs cannot meet these criteria and the purpose of the evaluation is to 
determine the cause of observed outcomes',- the consultant might encourage 

* 

the program to' consider developmental designs (Porges, . 1979) or single-' 
subject designs (Kratochwill, 1978; • MacLeod , Andrews & Grove, 1980).. 
Objective-based Designs. With the inclusion 'of IEP's in programs for * 
preschool handicapped children, the importance of objective-based evaluation 
•could increase dramatically. It appears logical and forthright to determine - 
the success of the program by assessing the accomplishment of children iri 
relationship to the goals and objectives stated in their IEPs. Caution ■ . 
should be given in suggesting the use of this approach, however, 'to include 

j 

procedeires that would demonstrate that the IEP objectives themselves are 

In- 
appropriate and valid, and that adequate data are available' to determine 

if the objectives have been achieved.. - 

Systems Models. There are few examples of the use of systems models for / 
eva|pation in programs for preschool handicapped children. The reasons 
for this could include lack of information concerning these models 'by both 
.evaluators and program staff, *br the very comprehensive and therefore 
seemingly unfeasible task of implementing such a design. Lack of use, 
however, does not indicate that these design!* or portions of them would 
not.be useful to HCEEP projects. For evaluations which are decision- % / 
oriented or require a thorough description of the program as it was 
implemented, a systems design might be useful. Aspects of models such as 1 
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' th'e process evaluation portion of Stuff lebeam's CIPP model might be 
particularly useful to staffs that wish to obtain evaluation information 
regarding their procedures. 

'Naturalistic Designs. .Quite a few projects would describe their evaluation 
methoftlogy as that of the case study or naturalistic type. In most cases, 
what is provided is a description of what occurred during the program with 
* special emphasis on the outcomes for individual children. Evaluators can 
assist program personnel in developing naturalistic designs which are 
methodologically sound. This would require a stricter adherence to procedures 
for conducting case or naturalistic studies. It would also require the 
development of schema which would allow for the interpretation of the 
p^gress of children or impact of the program from leng'thy descriptions. 
The evaluaior can be particularly helpful in refining and streamlining 
strategies to make this approach feasible. * 

Instrumentation; Perhaps .the question most often asked of the evaluation 
consultant in planning evaluations is "What instrument should we use to" " 
measure this?" A variety of resources which identify and review standardized 
and criter^n-reference<l-in&t^imeiits^^e_Jlsled^n Appendix "A. Other, 
instrumentation sources are those programs and projects in which more 
informal instruments have been developed (Cox, Patten & Trohanis, 1978). 

In assisting projects to select instruments ' it would seem important 
to encourage the use of criteria such as a.) appropriateness for the • 
individuals who will be assessed; b.) appropriateness for measuring the 
effect of die intervention; c.) Characteristics of the program; d.) the 
evaluation .questions posed in the evaluation design; e.) the information ** • 
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needed; and f.) skills of the staff in administering the instrument. 
' . ' While Project staff members should be encouraged to use the most. 

• valid and reliable instruments available to them, the consultant should 
- be prepared- to caution them regarding the limitations of existing 

instrumentation. Few instruments have been developed and validated for 
- handicapped children and it is often difficult to find ones whichSft^tch 
the goals of an innovative* model program. Help should be given in 
selection of the most appropriate instruments, with concommitant assistance . 
in identifying the limitations for use anf interpretation of resulting data.' 

For those instruments which must of necessity be developed locally, 
consultants should assist in planning for refining and field testing the 
instruments. While a complete two-to.-three year instrument development 
. program may not be feasible ok appropriate for a pr/ject , /basic content and 
format reviews and field tests should be^nducted so that^heVogram will 
^ have an indication that tfie instruments reasonably stable -and reliable! 
Jata Analysis . Consultants should assist staff members in determining that 
the data analysis procedures match the types of data and ( the types of questions 
that are^eing asked. In addition, they should encourage the use of procedures 
that are feasible for the program staff to conduct (or have conducted for 
thein) and be interpretable by them. This may require the consultant to V 
ascertain such things as the statistical analysis expertise of the persons 
involved in'the evaluation and the availability of 'computer's, computer 
• programmers, data coders, etc. For those programs In which statistical 
analysis expertise is not available, the consultant may need to suggest 
more descriptive than analytical data analysis techniques. 
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Criteria. It^is necessary to establish criteria against which to interpret 
evaluation results." A comparison of criteria with evaluation results 
permits the evaluator and^r staff to judge the effectiveness of the 
project. Criteria are usually considered in two domains: statistical 
significance and programmatic significance (Tallmadge, 1977). Statistical 
significancl^obtained by the use of statistical tests (Kerlinger, 1973). 
. Programmatic significance is.more difficult to specify, but of great 
inportaj^e in assessing the benefit of a project to children, parents, 
staff, etc. It may be necessary t# advise the staff to go^to several 
sources for establishing programmatic criteria. Among possible information 
sources are the! general base'of knowledge- in the area, prior experience, 
expert opinion, and even ."best guesses" in the case of very' new endeavors. 
Planning for Re port in g and Using the Results of the Evaluations 
; Perhaps one of the most useful services an evaluation consultant ?can 
perform is to assist project staff in identifying ways in which' they can 

i * 

use the results of their evaluation. This requires skills in the area of , - 
communication and staff development. It includes considering: a.) the 
purpose f^or reporting or ''using the results; b.) determining Ihe audiences' 
who will be receiving the results; c.) identifying which information or 
results will be used; and d.) determining the methods in which the Results 
will be shared and used. . c- 

Much of the planning in this area will already be done if .the evaluation- 
has been adequately focused. Evaluators and program personnel will know, 
a't least in a general way^howdie results^!: the evaluations will be used 
and reported. Planning fin this area, 'therefore, becomes a process of 
determining which specific results will be included and selecting the 



particular methods for reporting or using these results. 

It is at this point that planning schema such as that developed by 
> Trohani-s (Suarez and* Vandiviere, 1978) are useful. A helpful service ' 
that evaluation consultants can provide to projects here is to point out 
the variety of ways in which evaluation results can be reported and used. 
An executive summary or brief abstract of the evaluation results is a very 
useful document t^o have at hand. Brochures, slidetape presentations, and 
graphic handouts at, staff meetings could be suggested in addition to 
technical reports. f ~ / 

Planning for the Implementation of the Evaluation 

Evaluators and program staff often believe that the task of planning 
an evaluation is complete when the design is determined, questions are 
•identified, and data collections procedures are selected /and .planned. 
Project personnel left with only these portions of a plan often experience 
considerable difficulties in accomplishing the evaluation tasks. We 
believe that evaluation plans which do not include a plan for implementation 
are incomplete. Program staff need, to know who will conduct the evaluation, 
what resources are available, when things need to be accomplished, and to 
have some ideas how all of the activities will be managed and monitored. 

Usually, all staff members must pitch in and contribute to the process 
of evaluation. In order "to insure that the tasks are accomplished and to 
eliminate confasion, it is very important to identify the roles and 
responsibilities of each staff member and to secure' their commitment to the 
process. This would include the very important identification of the one 

person who will manage and coordinate the evaluation. It would also include 

v 

identifying those within or external to the project who might be involved. 




Evaluation is not a cost free ,epdeav©r (unless you consider it 
philosophically as does Scriven [1974]). Program staff may need^ assistance 
in identifying and targeting resources for such items as consultant \ 
assistance, purchasing instruments,^ copying costs, Space to work, clerical 
support, supplies, data coding, computer time and other similar items. 

The schedule for the accomplishment of the evaluation needs to be 

incorporated into the schedule for the overall program operation.' Key 

' i 

dates need to be identified for evaluation tasks and should be incorporated 
into the overall project timeline. 

While it may be inherent in both the identification of'roles and 
responsibilities and in scheduling, it ( is important that the project director 
share in or take the responsibility for the ways in which the evaluation 
will be managed and monitored. For the consultant, this may include 
assisting in the development of. contracts with outside 'consultants 
and providing ideas about how such contracts 1 can be managed. It should be 
clear to all of the project staff that the person who has the responsibility 
for managing the evaluation has the authority to ask for the accomplishment 
of scheduled tasks from other members of the staff. 

In summary, a final caution would seem to be in order. Evaluations 
of HCEEP programs should be within the scope of what is of most interest 
and importance and what is most feasible for the staff within the limits 
of available resources. It is very easy for the progrlnn staff and even 
evaluatprs to be carried away with plans for providing information in great 
depth about every aspect of the program. Evaluation should be a helping 
supplement to a program. It should not e!{id up^being an effbrt that is 
-almost... as great as the effort to operate the program itself. One of the 
more interesting and sometimes difficult tasks for -the evaluation consultant 
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is, then, restraining evaluation so that itj addresses the most important 
topics in a manner which can be accomplished well. 
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c- Figure 4 

PLANNING GUlfe FOR THE EVALUATION 
. OF EDUCATIONAL PROGRAMS FOR YOUNG CHILDREN AND THEIR FAMILIES 

> 

Tanya M. Suarez 
Associate Director for Evaluation # 

I nt roduc t ion 



The" "Planning Guide" is designed to provide a framework for planning evaluations for programs for 
young ch.Jdren and the i r fami 1 li es. It does this by posing a trie's of quest ions In four areas 
u n eva Cat lot resits ' and ft dete ™ ]n [ n * the . s P ec j fic evaluation design, (3) communicating and 
a clear oortra?? of til ? ( J- ,m P ,ement,n 9 the evaluation. Answering the questions should provide 
iu« h !i' ? evaluation for tha progra m staff and other interested individuals. It should 

also enhance the user's capability to conduct evaluations and use their results 



Using the Guide / 1 



for J h l V- J variety of ways in which the "Gui de"can \>e used. 'First, it can provide a structure 

n the nlTt fZ LS TL™ " ^ T^' 0 "" lndividua ' P« ans need-not be in the format suggested 
n the next fouc pages; however, they should, whenever possible, reflect consideration of the questions 
in a 1 sections. Second, parts of the Guide can be used to plan supplements 'to an existing I ? 
developing a management pi an to accompany an existing evaluation design. Finally, the Gufdecan'be ' 
used as a checklist against which an existing plan can be compared for completeness. 

Included wfth the "Guide" is. a list of references categorized by the major topical areas presented 
They are included to provide the reader with sources of additional information which miyue referred to' 
in mak.ng decisions regarding evaluation plans. Y rere rred to 
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Major Concern I. EVALUATION FOCUS: What should the evaluation accomplish? 



Purpose 



Why is the evaluation being 
conducted ? 

e.g., to meet funding require 
roents, to monitor/administer 
the project, to Identify and 
Improve weaker components, to 
strengthen the Information 
available for dissemination/ 
competition for funds, etc' 



d. 



Audiences 



Who are the audiences for the 
evaluation? 

e.g., project staff, funding 
agency, administrative agency 
community agepcles/leaders , 
professionals*, parents, e£c. 



a. 



O * Suarez, TAOS, UNC-Chapel Hill, HC 1980 
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Audience Information Needs 



What do the audiences need to 



know? 

e.g., progress of children, 
quantity and/or quality of 
services to children and fam- 
ilies, attitudes of children 
and/or families, program 
costs, etc. 



Key Program Components 



What are the key components 
of the program? 



e.g., services designed to 
increase the cognitive so- 
dal and motor development 
of children, services de- 
signed to Increase ^parents' 
skill 1n teaching their 
children, ^curriculum de- 
signed to Increase cognitive 
development of children, a 
service which links children 
and families with assistance 
available in the community, 
etc. 



c. 



Statement of Intent 



It ts the purpose of this 
evaluation to 



(why it 1s being done) 



i 
I 



by providing information re- 
garding 



(key components and audience 



needs) 



to 



^audiences ) 
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Major Concern II. 



Evaluation Questions 



Evaluation Design 



What information will the 
evaluation seek to provide? 



e.g., Have children made ex- 
pected progress during their 
participation 1n the program? 
Are parents better able to 
tefcch their children? Have 
Interactions among children 
and primary caregivers Im- 
proved? Are attitudes of 
parents, teachers, other re- 
lated agency personnel toward 
the program favorable? etc. 
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What design or set of proce- 
dures will be used to gather 
the information? I 



e.g., Pre-post assessment of 
the development. of children 
in experimental. jNpd contrql 
groups, monthly assessments 
of children's progress* on 
IEPs, survey of parental at- 
titude at end of program, 
observation of mother/child 
interaction before, during, 
and after training, etc. 



b. 



ERIC 
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PLAN: What should be the major components of the evaluation? 



Data Collection Procedures 



What instruments or forms 
will be used to gather and/ 
or record the information? 



e.g., McCarthy Scales of 
Children's Abilities , staff 



developed attitude scale to 
"measure parental attitude 
toward the program, Caldwell 
Home Inventory , staff de- 



veloped form for recording 
parental participation, etc. 



C. 



Data Analysis 



How will the resulting data 
be analyzed? 



e.g., Correlated t-test of 
difference between pre- and 
post-test means, comparison 
of expected level of develpp- 



7 7~ ; v ' ^ui»i H M3iicu, par LiuipdL ion OT 

ment (determined using a re-llf* of the parents in two or 

nt*acc^An inilui>io\ i.UL a.. . . • _ . ..... 



gresslon analysis) with actu 
al level of development fol- 
lowed by a test of the signi- 
ficance of the difference, 
computation of percentages $of 
favorable responses, computa- 
tion of frequencies of parti- 
cipation, etc. 



Evaluation Criteria " 



How will the results be judged 
or interpreted? 



e.g., Statistically significant 
difference -between means (.05 
level), 752 of objectives ac- 
complished, participation of 




more project activities, etc. 



c. 



d. 
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Major Concern III. COItlUNICATING/ UTILIZING PLAN: How will the 'results of the evaluation be comnunicated and used? 



Purpose 


Audiences to be Address*^ 

9 


Results to be Used 


* ■ • - ~ •* 
Method of Utilization * 


Why are the results of the evaluation 
beinq communicated and/or used? 


What audiences will be qiven and/or 


What Information and results will be 


In what ways-ftUT results be 
shared and u^ed? . 

e.g., distrlbutfon of print mate- 
rials such as reports, articles, 
etc.; presentation of.-non-print 
media such as s'l^detape, video- 
tape, TV/rad1o, etc.; personal 
contact, tra1ning,/etc. 

* « 

a. 


REFER TO PURPfKPS PHP fvaihattow arJ4 
e.g., reporting to fundina/adminls- 
Liaiiye ayencitfbi parents, aovisory f 
boards, etc., program planning, staff 

rlpW m 1 nrVTWrit" /~ Arm** i n i f i nn „ *. — 

-uev v \ uptncit i , LUtiiTiun ica Lion, etc. 
COMMUNICATION 

a. ♦ 


use the resul ts? 

REFER TO AUDIENCES FOR EVALUATION 
a. 


communicated and/or used^ 

e.g., aj 1 results, child progress ' 
data, monitoring data, cost infor- 
mation, ~ etc. 

- 

a. 




♦ 






b. 


b. 


b. 


b. 


* 








• 

UTILIZATION 
a . 


a. 


T 

* 

a. 


* 

* » 

i 


* — — 


— 1 








b. 


b. 


b. ! 











I 

I 
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h ^ 0r Conc * rn »V. IMPLEMENTATION PLAN: How will the evaluation plan be. accompl ished? 



% , Personnel Role Spfeci f Ication 
« 



Resource Allocation 



Scfciedul ing 



Monitoring 



Who will conduct the evaluation ? 
a. Manage/Coordinate: 



b. Design Evaluation: 



d. Collect data: 



Analyze data: 



f. Write summary reports: 



g. Other: 





What other resources are available 
for the evaluation? 



* c. Select/Develop Instruments: 



a. Consultant Assistance: 



b. Materials: 



c. Facilities/Space: 



d. Clerical Support: 



What are the key dates on which 
tasks need to be accomplished? 



$ Amount 
or N/A 



Date 



Supplies: 












Computer Time: 


Other: 


\ 





TOTAL FINANCIAL RESOURCES 
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a. Evaluation Plan : 

b. Selection/Development of 
Instruments : 



1. 
2. 
3. 
4. 
5. 



c. Data Collection; 



s 



4. 



d. Data Analysis: 1 

e. Report Writing: 
f Other: 



How will the evaluation be mom to red? 



e.g., regular staff meetings, regular 
meetings of Manager/Coordinator with 
persons responsible for specific tasks, 
review of quarterly submissions of 
information gathered to date, etc. 



c. 



■^v" CO 



J 



JDRP - THE JOINT DISSEMINATION AND RE\(rEW PANEL - AND HCEEP 
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JDRP - THE JOINT DISSEMINATION 
AND REVIEW PANEL - AND HCEEP 

/ 



Introduction 



_ Many HCEEP projects are interested in the Joint Dissemination ,and 
Review Panel as an audience for their evaluation results. This section 
presents information .on- the nature and purpose of JDRP, and outlines some 
guidelines for assisting project staffs in deciding if presentation to 
the Panel is appropriate for them and their program. A description of 
procedures for submitting information" to JDRP through the Office of 
Special Education also is included. 

The Purpose of JDRP 
The Joint Dissemination and Review Panel (JDRP) was organized in 1972 
by the U.S. Office of Education (USOE) and was enlarged in 1975 to include 
representatives from the National Institute for Education (NIE) . it is 
currently sponsored by the Department of Education, its purpose is to 
assess the effectiveness of federally funded* programs for the purpose of 
dissemination of information regarding them by the Department. If the 
JDRP concludes, on the basis of evidence of effectiveness presented to 
them, that an educational program is, indeed, effective, the program is ' 
said to be "validated." This validation is retired before information 
regarding programs can be di^eminatTd~by. the federal government. 

The Panel, which is comprised of 7 members selected randomly from a 
group of 22 people representing various disciplines within educational 
research, meets periodically to review program submissions. 'Panel members 
are selected on the basis of their ability to analyze evaluation-based 
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evidence of effectiveness of products and practices, and on their general 
experience in education and research, 
* 

~ * ~ Guideline s for Decision Making Regarding JDRP 

It is critically important for an HCEEP project to determine whether' 
or not the JDRP is a potential evaluation audience early in the evaluation 
planning process, so that adequate data can be collected during its 
three-year funding period. A consultant can be very .helpful in assisting 
projects with this decision. 

Four major questions must be considered in the decision: ' 
1.) Will JDRP approval benefit the projefct? 

2-) ^p*n the model be replicated? J ■ ' 

3. ) Is the evaluation design adequate? 

4. ) Are the £vailable expertise and resources adequate? 
Will JDRP Approval Benefit the Project ? 

As mentioned ^previously,, JDRP approval permits the dissemination of 
information regarding ajroject or program by the federal government. The 
vehicle for this dissemination is the National Diffusion Network (NDN) , 
Programs validated by the JDRP are described in a book, Programs That Work, 
which is distributed widely around the country. Federally-sponsored 
"facilitators" in each state have the responsibility of acquainting local 
school personnel with information regarding JDRP approved programs. 
Finally, funds are available from NDN through competitive grants, for the * 
dissemination, demonstration and replication of a program by its own staff. 
JDRP approval provides therefore: 1.) national recognition; 2.) automatic - 
national dissemination of information regarding a program; and 3*.) the 
opportunity to apply for funds to disseminate, demonstrate, and/or replicate 



the program. 
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The Officf of Special Education also provides special opportunities 
for JDRP validated projects. While regulations are not finalized, .the • 
Office is beginning to consider JDRP approval as a "necessary but not 
sufficient" criterion for the acquisition of HCEEP Model Outreach (formerly 
Outreach) funds- after the three years of demonstration funding. The 
purpose of thes^e funds is to stimulate more and better services for young 
handicapped children and their families. Activities which may be funded 
are similar to those funded by NDN. They include: a.) incre^ng awareness; 
b.) product development and distribution; c.) stimulating replication sites; 
d.) stimulating' state involvement; e.) training, and f.) other specific 
consultative acuities (Swan, 1978). jdrp approval> therefore> may giye 
a project a completive edge for acquiring additional funding f r0 m HCEEP 
for dissemination and demonstration activities. 

National recognition and the opportunity to apply for funds for 
dissemination and demonstration are the obvious benefits of' JDRP approval. 
If the project staff are interested in these things, consideration of a 
submission to JDRP is appropriate. The consultant should explore the real 
time and effort requirements of a JDRP submission and help the staff reach 
a decision regarding the appropriateness of a submission for their needs. 
Can the Model Be Replicated ? 

Two major factors should be considered in helping projects assess, the 
replicability of their model-1.) its level of documentatlon and model 
description and 2.) its broad replicability. The -project must be developed 
and described clearly and thoroughly, so that others may replicate it with 
f ldrfltTto the model. It must also be replicable by a different staff 
at a different location with different clients. The .evaluation consultant 
may help <the pro j ict staff re ^ ew th £ Q ^ ^ detemine ±f ^ 



project is appropriate for JDRP. 
Is the Evaluation Design Adequate ? 

The Joint Dissemination and Review Panel IDEABOOK (Tallmadge, 1977) 
presents a discussion of the use of evaluation designs recommended for 
JDRP submissions. Since HCEEP projects are seldom, if ever, able to use 
# true experimental designs, the section on quasi-experimental and other 
designs is of the most interest. « Tallmadge lists six methods of estimating 
"without-intervention" conditions 'for use in planning the design (listed 
in descending order of preference and credibility): 

. a highly similar but non-equivalent control group 
. historical (pre-intervention) data 

. a comparison group formed by dichotomizing an originally intact 
. group, into treatment and comparison components around some pretest 
cutoff score (Regression-discontinuity and regression-projection 
models. See Campbell and Stanley, 1963; Horst, Tallmadge, and 
Wood, 1975.) y 

» 

• national, regional, or local normative* data 
. logic 

. expert testimony 

Additionally, the use of single subject designs and Goal Attainment 
Scaling have become alternatives for evaluation designs. 
Are the Av ailable Expertise and Resources Adequate ? " 

If, after reviewing the previous questions, JDRP seems to be an 
appropriate evaluation audience, it is crucial to review, again, the project 
resources. Preparing for JDRP requires a . considerable investment in project 
staff time for such activities as planning the evaluation, administering 
tests, taking daily or weekly data, keeping careful records, analyzing data, 
interpreting results, and developing the JDRP submission. In addition, if 
^control or comparison group is to be used in the evaluation, much 



-33- 



administrative time is required in agency liaison, planning testing 
'schedules,- communication, administering' tests, travel, and reporting * ' 

test results back to the cooperating agency. 

The project director or another staff member also must possess 

considerable research and data analysis skill, if these skills are not 

available within the project, dollars and time must be spent with an 
external evaluator who can actually conduct the evaluation of program 
impact and interpret the results. There must be budgetary or in-kind' 
resources for purchasing tests, administering and scoring tests, key- ' 
punching, computer time, and typing and reproducing reports. Finally, ' 
staff time must be available for writing and preparing to deliver the 
submission. It is clear that a considerable effort may be required to 
meet the criteria for JDRP review of evidence of effectiveness. 

OSE Procedur es f or Submission Development and Presentation 
Consultants are frequently asked to provide information concerning 
the process for preparing and presenting" JDRP submissions. The submissions 
are prepared by the project with the appropriate OSE project officer and 
submitted to the panel by the -project officer. Projects should, therefore, 
be f encouraged to contact their OSE project officers for this type of 
information. At the time this Handbook was prepared, Figure 5 represented 
the basic process for developing submissions, as developed by OSE. 

As Figure 5 demonstrates, the submission development process is one 
which takes place between the project staff and OSE project officer. It 
is in the communication between the two that the 10 page submission is 
developed. Evaluation consultants to projects most often assist in: 
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Figure 5 

JDRP SUBMISSION DEVELOPMENT AND PRESENTATION PROCESS 



Project Director* Discusses 
Developing Draft Submission 
for JDRP with P.O. 



Project Director Submits Draft 
of JDRP Submission to P.O. 



P.O. Receives Draft Submission 
and Requests Critique 



Person(s) Selected to Critique 
Draft and Make Recommendations 
to P.O. 



P.O. Communicates with Project 
Director and Indicates Submis- 
sion Will Be Forwarded to JDRP 






P.O. Writes Memo to JDRP 
Through DID Director and OSE 
to Request JDRP Scheduling 


9 1 




P»0. Maintains Continued 
Communications with Project 
Director Notifying of 
Scheduling Date 






.P.O. and JDRP Coordinator Meet 
with Project Director et al.« 
to Discuss Submission .Verbally 
at least Tfrree Hours Prior to 
JDRP Presentation 


1 * 




Project Director, 
Present to JDRP 


et al. , 



WWS July 1980 



Project Sends Sample 
Set of Child Progress 
Data to P.0. 



Project Receives Approval 
from P.O. to Develop Draft 




P.O./Critiquer Commun- 
icates with Project 
Director Concerning 
Recommend a t ions 



Project 
Director 
Revises Draft 
and Resubmits 



f 
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a,) helping the staff determine if they will seek JDRpUalidation; b.) 
designing and implementing the project's evaluation; and in the case o 

-consultations which are not sponsored by TADS, 7 c.) assist the project 

y 

director in preparing and presenting the results. 



I* ™» ° f 0SE P ° liCy that ^-sponsored consultations related 

to JDRP NOT include the development of the submission or presentation 
of the evaluation results- -to-che Panel. 



SUGGESTIONS FOR EVALUATION CONSULTANTS: 
PROCEDURAL. AND 'PERSONAL 



'Introduction 



9 

In addition to knowing the context in which evaluation takes place , 



and some of the specific evaluation needs of the HCEEP demonstration projects, 
0 .consultants need to\knol and use some of the 'basic principles of effective 

consultation.. These are reviewed herein discussions 'of the consultation 
* process, the stages in a consultation and helpful hint^pthered from the / 

TADS staff and consultants .who have worked with these projects. 




Providing Evaluation Consultation: The Process 
Figure 6 outlines the stages of consultation, , as adapted (Suarez and " 
Vandiviere, 1980) f f om The Consulting Process in A^ti™ (lippitt and Lippitt, 
1978). ■ The consultation stages, are applicable to short-term consultations 
of 1 or 2. days; or to $ore "extensive consultations which extend over a period 
of months 01; yea^p^ , » 



s section should be read while keeping in mind two major principles 



1.) Successful evaluation consultation is oriented fo the unique needs 
or tnepto^pt, not to the theoretical orientation or biases of the 



t. • 




Successful' evaluation consultation produces evaluation plans which 
are related to project goals, are realistdc and feasible within the 
resources of the project. <j 

Stages of Consulta tion *' 

' 77. " * > t> 

-Consultation^/kssistance) can be viewed as having four major stages . 

whichSre soiffewhat similar^ the stages in evaluation planning: . Focusing ." 
Planning . Providing , and Follow-up . 



Fig! 

- PROVIDING AND RKCEIVINC ASSISTANCE IN EVALUATION: 
ANj| ADAPTATION OF fhF LIPPITT/MPPIT1 CONSULTATION MODEL 



ASS-ISTANCE STAGES 



MAJOR TASKS 



II. 



FOCUSING THE 
ASSISTANCE 

A. Contact/Entry 




B. Problem 

Identification 



C. Diagnosis 



PLANNING THE 
ASSISTANCE 

D. PlsfmUng for * 
Action 



Initial contact ^fe 
Orientation ' 

Explore need for evaluation assistance 
Explore readiness for assistance 
Explore potential* for working together 

Explore manner of communication 

Explore trust ' } 

Establish credentials 



Review program 

Identify general evaluation needs 
Identify needs requiring assistance 



Determine extent of need 
Determine priority of needs 
Determine desired outcomes 



Confirm appropriateness of working together 

(communication, trust, credibility) 
Clarify expectations 
Develop goals for the assistance 
Specify taaka 

Assign roles and responsibilities 
Schedule assistance 
Develop contract 



EXAMPLES 0F>6UCCESSFUL PRACTICE 



Consultants 



•Begin to become familiar with 
the program and its personnel 

•Determine client's experience, 
expertise, resources in 
evaluation 

•Determine client's purpose 
for evaluation 



■Obtain clear understanding 
of the program 

•Obtain understanding of 
evaluation needs 

•Identify, internal and External 
resources available for. 
evaluation; human 

mechanical 

financial 



•Ask clarifying questions 

•Provide organizational 
framework 

'Set parameters of the 
assistance *i 



Administrators 



'Acquaint consultant with program 
and its personnel 

t 

^Determine consultant's appropriate- 
ness for the assistance needed 

•Determine consultant's compatabil- 
ity with staff and program 

•Make personal commitment to involve- 
ment in the?assistance 



•Develop clear understanding of the 
program 

•Obtain understanding of evaluation 
needs , 

•Specify what is needed rrom the 
assistance * 



•Identify internal and external resources 
available for evaluation: human 

mechanical 
f inane lal 

•Determine staff responsibilities 



^""'^cltS/SIMso.' ? ' Pr0vldl "g and Receiving Assistance In Evaluation . A paper presented at the North Carolina Association for 



Research 



ERIC 



* ^W^ure 6 





ASSISTANCE STAGES 


\ 

MAJOR TASKS 


EXAMPLES OF SUCCESSFUL PRACTICE 












I IK 


PROVIDING/RECEIVING 
THE ASSISTANCE 




Consultants 


Administrators 




E. Preparation 


Reassess* plans re cirrent needs, 
modify plans if necessary 

Prepare for agreed upon activities 


•Read all available materials 

•Develop ^potential list of 
"areas of inquiry 


•Schedule; i.e., people, facilities 

♦Orient appropriate staff to purposes 
and schedule of the consultation 








•Pull information together 
•Prepare materials 


'Gather needed resources, materials,.. - 
etc. 


i 


P. Taking Action 

- 


Reassess plans re_ current needs, 
modify plans if necessary 

Provide assistance ' ' , 

Solicit and provide feedback 

Design continuity supports. 


•Observe program in action 

•Provide a variety of assis- 
tance 

•Provide assistance fbcused 
on the level of staff exper- 
tise and experience 

•Provide technical and pro- 
cess assistance 


•Provide access to entire program 
•Take leadership role with staff 
♦Actively involve staff 
•Determine feasibility of evaluation 
•Provide and solicit feedback 


IV. 


FGLLOW-UP 


* 

• 


•Solicit and provide feedback 
r- 






C Continuity 
and/or 
Termination 

- 


Clarify future tasks 

Assess needs for future assistance 

Plan for additional action or 
terminate 


'Provide d&tails of feedback 
given during consultation 

•{Provide specific recommenda- 
tion's f nr Fftl lnu*iin ar*t\\i\~ 
k i.uii3 i ui 1 \J 1. luw up aLLlVl ^ 

ties 


•Solicit feedback 

♦Assume* or assign responsibilities 
for next action steps 

•Evaluate level of attainment of goals 






• 


•Provide promised or supple- 
mental materials 


*AflA488 need for additional assistance 

4 * 








•.Provide recommendations that 
are feasible 

* 


'! 




• 






« 




i> • 
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Focusing the Assistance . The necessity for focusing the consultatioa 
activity itself parallels the need for" focusing in planning, the evaluation- 
it forms the core around which the work is developed. -Focusing consultation 
has three major stages: Contact; and Entry . Problem Identification , and 
Diagnosis . 

. During Contact/Entry the consultant and project work together to become 
acquainted with one another, to explore the project's actual need and 
readiness for assistance, and to explore the potential for working together. 
A communication system is developed, credentials and expertise are 
established, *nd <£he development ef mutual trust is begun"* The consultant 
begins to become familiar with the program, begins to learn about the. 
project's resources and experience and. expertise in evaluation. He/she ' 
begins to determine the project's major purpose for evaluation. The 
administrator acquaints the consultant with the program and begins to 
assess the consultant's appropriateness, He/she also makes a commitment 
to the evaluation at this point. (When the consultation is provided through 
TADS, the Technical Assistance Agreement between TADS and the project can 
be used as the planning springboard.) 

Problem Identification requires the consultant and'project to review 
the program, identify general evaluation 1 needs , and then highlight those 
needs where consultation can be most useful arid effective. The consultant 
and the administrator obtain a clearer understanding of the program and 
its needs during this time. 

The final step in focusing the consultation is Diagnosis —the 
. t i. ' ■— 

determination of the extent and .priority of the needs £or assistance, and 
establishment of desired outcomes for the consultation. The consultant - 
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begins to identify internal and external resources for the evaluation, and 
the administrator begins to specify what is needed from the evaluation. 
planning Assistant . ■ After the needs for help have been identify, 

priorities determined, and desired outcomes described, it is time'^o Plan 

f ° r ACtl ° n - The tasks in this P^se are very important to a successful 
* consultation. 

The consultant and the project staff leadership must first reconfirm 
that they can work together with good communication, in an atmosphere of 
trust and confidentiality and with an understanding and acceptance of 
. their .mutual expertise. r The consultant clarifies the expectations and * 
goals- for the consultation, specifies tasks to be done, discusses the 
assignment of roles and responsibilities and, if appropriate, develops 
a consultation contract. It is helpful if the consultant provides the 
organizational framework for the planning, and facilitates it through 
asking clarifying questions, re-stating questions and decisions, and 
setting parameters. As the planning progresses, the administrator begins 
to identify resources, determines preliminary staff responsibilities, and 
clarifies the lines of authority and decision making. 
'Providing the Assistance. Provisio*of the assistance has two stages: 
Preparation and Taking Action . 

If a significant amount of time has elapsed since the initial contact-, 
the consultant may carry out some Preparation activities.. It is important 
to reassess the current needs for help, and modify plans for the consultation 
if necessary. The consultant reviews' all available materials concerning 
the project, pulls the information together, and develops a potential list 
of areas of evaluation inquiry or questions regarding' the program. The 



GO 



administrator sets up schedules, orients staf/ 4nd begins to gather need 



resources. 



The planning and forethought which have been described to this po^nt 
ar-e then combined into the stage Lippitt and Lippitt call Taking Action' . 
During, this stage the consultant again reviews and confirms, with the 
administrator, the plans for the consultation, and then provides the 
assistance. He/she sees the program in action (if this has not occurred 
during planning),. and provides assistance in response to the needs of the 
project staff. The consultation is focused at the appropriate level of 
project expertise and experience, and includes both process and technical 
assistance. The entire activity is strengthened' if the consultant is 
careful to provide appropriate feedback to the project staff, and solicits 
feedback concerning his/her own work in order to keep it on target and 
useful. 

During a consultation the consultant may need to assume a variety of 
roles in order to accomplish the objectives of the consultation. Druian 
(no date) in an adaptation of Lippitt's (1973) work lists the following 
potential consultant roles: 

. Advocate: persuades client to proper approach 

. Expert: gives expert advice to client 

. Trainer: develops training experiences to aid client 

. Alternative Finder: provides alternatives to client 

. Collaborator: joins in problem solving 

. Process Helper: assists client in problem solving process 
. Resource Linker: serves to help client collect information 

. Catalyst: serves as- a catalytic agent for client in solving 
t^e problem - 
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These roles are arranged in order from greatest consultant/least 
client participation to least consultant/greatest client participation. 
Choice of role is dependent upon the ~need that is being addressed, the 
skills, expertise and interests of the client and the nature of the 
situation, in which the consultation takes place. 

The "final step in Taking Action is the identification of supports 
for continuity of the evaluation planning— the prelude to Follow-up . 
Follow-up. f During the Follow-up stage the consultant and project clarify 
future tasks, assess the need for future consultative assistance, and 
eicher Plan for " Additional Action or Terminate the consultation. The 
consultant, in either case, provides recommendations for follow-up" 
activities, provides materials for future use as requested, and provides 
a set of final recommendations resulting from the consultation. 

Summary 

Utilizing this framework for conceptualizing the consultation process 
should, we believe, help evaluation consultants make the best use of their 
expertise, and assure an orcjerly,. well-planned and organized consultation 
activity. We also are well aware that missed planes, blizzards, illness 
and other acts of misfortune can and do affect the best laid intents and 
procedures. These situations are best met with flexibility and imagination 
to produce a successful consultation. 

Finally, a few miscellaneous hints and cautions may be helpful. 

... During the consultation process, keep in touch with staff* attitudes 
toward the evaluation— nurture enthusiasm, create/transfer 'ownership 

... Remember your role— the project director is the final decision-maker 
concerning the evaluation. 

... Keep all information ia total confidence — never .gossip. 

... Be sensitive to situations which are beyond your competence or 
knowledge— don't hesitate to obtain assistance for yourself! 

G2 
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If you find yourself in a situation which violates or strains ° 
your ethical principles, terminate the consultation as gracefully 
as possible. 

Refer all questions of federal policy which cannot be answered 
from the Regulations to the OSE project officer. 

Finally, -*don f t promise moire than you can deliver, and keep your 
promises. 



c 
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, Selected Resources for Evaluators 



Following is a list 'of selected resources which may be of use in planning 
and providing evaluation IconsultWon. The references were selected 
because of their applicability to evaluations of demonstration programs 
for, young^ handicapped children and are, therefore, not intended to be 
exhaystive. References are included in the areas of: a.) evaluation 
planning and design; b.) measuring child progress; and c.) consultation. 



EVALUATION PLANNING. AND DESIGN 



I. Focusing the Evaluation ' * 

> 

Anderson, S.B., and Cole, _ C.D„ Editor's -'notes: . The expanding role 

program evaluation. New Directions for Program Evaluatio n. 1978. > 
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Chelimsky, ;F. Differing perspective! * of evaluation. - New Direct ions 
for Program Evaluation . 1978, (2), 1-18.' 
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Kniefel," T.M. {Suarez). Administrator's guide to evaluation in local 
school dist&ctis . Raleigh, N.C.: N.C. Department of Public 
Instruction, 1975. 



Morris, L.L.., and Fitz-Gibbon, Cg. The program evaluation kit." 
Beverly Hills: Sage, 1978. 1 

Patton, M.Q. Utilization-focused evaluation .^ Beverly «ills: Sage-, 1978, 

Rutman„L. Planning an evaluation study. In^L. Rutman (ed.), Evaluation 
.research" methods : A basic" guide . Beverly Hills: Sage* 1977"; 

Suarez, T.M;, and' Vandiviere, P. (eds.) Planning for evaluation : A 
resource book for programs for preschool handicapped children - ' h 
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